Honeypot #21 將 Cowrie 輸出到 ELK stack - ELK Stack 環境安裝 - iT 邦幫忙::一起幫忙解決難題，拯救 IT 人的一天

2022 iThome 鐵人賽

Security

HONEYPOT×TRICK 100%不純保證系列第 31 篇

Honeypot #21 將 Cowrie 輸出到 ELK stack - ELK Stack 環境安裝

14th鐵人賽

魯大常

2022-10-18 09:03:01

1373 瀏覽

分享至

雖然沒有時間的壓力，不用水文章了。不過基本上還是按照一日的份量來寫文章，這樣應該也比較會有鐵人賽的意義，每一天學習，學習一點點

官方文件在輸出這一塊的著墨很多，尤其在日誌管理這一塊，日誌在大部份的管理軟體也都用的到。所以這一次的主題就放在 ELK Stack 的學習上。

官方文件的指引是使用 ELK 7.x 與 Java 8。這裡的測試環境會試著使用新版本，因此先參考elastic 與 JVM 搭配的文件，發現 Java 的影響很大，而又由於 Java 版本的區別，長期支援版為 8 > 11 > 17，最後就選用最新版的 Elastic 8.4.3 與其搭配的 Java 17.

環境:

Debian 11
Java 17
ELK stack 8.4.3

安裝軟體

安裝軟體分為前置軟體與 ELK stack 本身。只要有選擇配合的 Java 版本，在安裝上基本沒有太大的問題。

安裝 JAVA 環境

安裝 openjdk 17 版

sudo apt install openjdk-17-jre

安裝後可確定 java 的版本

debian@debian:~$ java -version

openjdk version "17.0.4" 2022-07-19
OpenJDK Runtime Environment (build 17.0.4+8-Debian-1deb11u1)
OpenJDK 64-Bit Server VM (build 17.0.4+8-Debian-1deb11u1, mixed mode, sharing)

加入 elastic 軟體來源

先加入 eleastic 的軟體來源

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

再進行 apt 更新

apt-get update

安裝 ELK stack

這裡開始正式安裝 ELK 的相關套件。
先是安裝前的下載檔案的相關軟體。

sudo aptinstall apt-transport-https wget

再來就是安裝 ELK 的組件，elasticsearch,logstash,kibana

sudo apt install elasticsearch logstash kibana

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  elasticsearch kibana logstash
0 upgraded, 3 newly installed, 0 to remove and 38 not upgraded.
Need to get 1,184 MB of archives.
After this operation, 2,435 MB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 elasticsearch amd64 8.4.3 [566 MB]
Get:2 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 kibana amd64 8.4.3 [285 MB]
Get:3 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 logstash amd64 1:8.4.3-1 [332 MB]
Fetched 1,184 MB in 1min 24s (14.1 MB/s)
Selecting previously unselected package elasticsearch.
(Reading database ... 171599 files and directories currently installed.)
Preparing to unpack .../elasticsearch_8.4.3_amd64.deb ...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Unpacking elasticsearch (8.4.3) ...
Selecting previously unselected package kibana.
Preparing to unpack .../kibana_8.4.3_amd64.deb ...
Unpacking kibana (8.4.3) ...
Selecting previously unselected package logstash.
Preparing to unpack .../logstash_1%3a8.4.3-1_amd64.deb ...
Unpacking logstash (1:8.4.3-1) ...
Setting up elasticsearch (8.4.3) ...
--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : GMF_76jYmz83y8BK97dg

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

-------------------------------------------------------------------------------------------------
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
Setting up kibana (8.4.3) ...
Creating kibana group... OK
Creating kibana user... OK
Created Kibana keystore in /etc/kibana/kibana.keystore
Setting up logstash (1:8.4.3-1) ...

以及非必要，但官方仍建議的 log 處理軟體 filebeat

sudo apt install filebeat

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  filebeat
0 upgraded, 1 newly installed, 0 to remove and 38 not upgraded.
Need to get 61.2 MB of archives.
After this operation, 275 MB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 filebeat amd64 8.4.3 [61.2 MB]
Fetched 61.2 MB in 11s (5,615 kB/s)
Selecting previously unselected package filebeat.
(Reading database ... 241516 files and directories currently installed.)
Preparing to unpack .../filebeat_8.4.3_amd64.deb ...
Unpacking filebeat (8.4.3) ...
Setting up filebeat (8.4.3) ...

安裝網頁伺服器

sudo apt install nginx apache2-utils

啟用服務

sudo systemctl enable elasticsearch logstash kibana filebeat nginx

Synchronizing state of filebeat.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable filebeat
Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable nginx
Created symlink /etc/systemd/system/multi-user.target.wants/elasticsearch.service → /lib/systemd/system/elasticsearch.service.
Created symlink /etc/systemd/system/multi-user.target.wants/logstash.service → /lib/systemd/system/logstash.service.
Created symlink /etc/systemd/system/multi-user.target.wants/kibana.service → /lib/systemd/system/kibana.service.
Created symlink /etc/systemd/system/multi-user.target.wants/filebeat.service → /lib/systemd/system/filebeat.service.

相關軟體的設定

安裝好軟體後，基本上將 cowrie 內建的各配置檔複製一份到各軟體即可。差別只在新版的有些設定值會與官方文件不同，要留意一下。

設定 ElasticSearch

修改 elasticsearch 設定檔，取消 xpack 的安全設定

sudo nano /etc/elasticsearch/elasticsearch.yml

# Enable security features
#xpack.security.enabled: true
xpack.security.enabled: false

不加這行的話，在之後使用 curl 測試時會出現「Empty reply from server」

啟動 elasticsearch

sudo systemctl restart elasticsearch

測試成功後會返回 JSON 的訊息。

debian@debian:~$ curl http://localhost:9200

{
  "name" : "debian",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "ZXIOhHSzSxCsp-GcFIlbDg",
  "version" : {
    "number" : "8.4.3",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "42f05b9372a9a4a470db3b52817899b99a76ee73",
    "build_date" : "2022-10-04T07:17:24.662462378Z",
    "build_snapshot" : false,
    "lucene_version" : "9.3.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

設定 Kibana

建立 kibana 的 log 目錄跟權限

sudo mkdir /var/log/kibana
sudo chown kibana:kibana /var/log/kibana

在本次的新版可跳過此步驟，上述的安裝過程已經有此目錄，跟適當的權限

修改設定，可以都保留不變

sudo nano /etc/kibana/kibana.yml

#server.host: "localhost" (預設即可)
#server.name: "your-hostname" (可變動)
#elasticsearch.hosts: ["http://localhost:9200"] (預設)
#elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"

新版已沒有 logging.dest。已變更為 fileName，且預設已是 /var/log/kibana/kibana.log

設定 Logstash

必須先註冊 Maxmind 的 GeoLite2，登入後從左下角的 Download Files > 找到 GeoLite2 City ，下載 Download GZIP，解壓縮後會得到一個「 GeoLite2-City.mmdb」，再執行以下指令

sudo mkdir -p /opt/logstash/vendor/geoip/
sudo mv GeoLite2-City.mmdb /opt/logstash/vendor/geoip

複製 cowrie 的設定檔至 logstash 中

 sudo cp /home/cowrie/cowrie/docs/elk/logstash-cowrie.conf /etc/logstash/conf.d

重新啟動 logstash

sudo systemctl restart logstash

設定 FileBeat

複製 cowrie 的設定檔至 logstash 中

sudo cp  /home/cowrie/cowrie/docs/elk/filebeat-cowrie.conf /etc/filebeat/filebeat.yml

修改 cowrie 的 log 位置。

sudo nano /etc/filebeat/filebeat.yml

  paths:
    #- /home/axelle/cowrie/var/log/cowrie/cowrie.json*
    - /home/cowrie/cowrie/var/log/cowrie/cowrie.json*

重新啟動 FileBeat

sudo systemctl start filebeat

測試 (Kibana)

Kibana 即為 ELK static 中負責圖形介面的軟體。測試是否能透過 logstash 抓到 cowrie 的log。

curl 'http://localhost:9200/_cat/indices?v'

health status index                             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   cowrie-logstash-2022.10.17-000001 KZ2ZB-fQTkSq4yFv1Q4WBw   1   1        382            0    315.7kb        315.7kb

再確認看是否正確

curl -XPUT 'localhost:9200/cowrie-logstash-2022.10.17-000001/_settings' -H "Content-Type: application/json" -d '{ "index" : {"number_of_replicas" : 0 } }'

{"acknowledged":true}

注意 cowrie-logstash-xxx 要替換成上一個指令顯示的日期

最後連上網頁 http://localhost:5601/app/management/kibana/dataViews/ ，按下「Create data view」上加入 cowrie 的 logstash:

name: cowrie-logstash (自取)
index pattern: cowrie-logstash-*

再按下「Save data view to Kibana」即可。

至此完成 cowrie 與 ELK static 的連結，但要如何呈現，這些分析的圖表還沒有建立出來，下篇待續...

參考資料

Debian 安裝 java 17
ELK與其故支援的 JAVA
elasticsearch 關閉 Xpack

HONEYPOT×TRICK 100%不純保證系列第 31 篇